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MEMORANDUM FOR: DD/Pers-P&C 
; DD/Pers-SP 
DD/Pers-R&P 
FROM . > F. W. M. Janney 
SUBJECT : Effectiveness Indicators 


REFERENCE : Report of OP Task Force on Effectiveness Indicators 
‘June 1976 : : 


1. As the new Fiscal Year is underway, this is to remind you that 
the system of effectiveness indicators is now operative. During finai. 
discussions of the task force, it was decided to eliminate Indicator 
II-C, Alcohol Problems, which is now an OMS responsibility. It was also 
decided to modify Indicator IV-C to measure the number of employees 
taking optional insurance rather than the amount outstanding per employee. 


2. I shall expect to teceive reports on the following schedule: 


INDICATOR I - A.¢ Recruiting - Semiamnually 
B. EEO Recruiting - Annual 


Ii - A. Reassignment - Annual 7 . 
B. Retirement Benefits Counseling - Annual 
C. Eliminate . 


tii - A. Position Grade Control - Annual 
B.. Review of Positions - Annual 


Iv - A. Employee Perceptions - Following survey 
’ - By Level of Effort Indicators - As appropriate 
C. Market Penetration (Insurance) - Annual om 
V - A. Errors (Pers. Record-Keeping) - Monthly 
B. Processing Time of Transactions - Monthly 


C. Timely Statistical Reports - Monthly 


ae VI - Internal Management - As PDP and APP data become 
available 


3. Please send information copies of these reports, when made, to 


Chief, Plans Staff, OP. 
: STATINTL 


s . 
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INTRODUCTION 


a On instruction from the DDA, the Office of Personnel created a 
task force in the fall of 1975 to "devise systems and procedures to 
develop and establish a‘series of positive indicators against which 
program missions (functions) can be judged." 


The task force was eunrise of representatives from each of the 
six program areas in the Office of Personnel. 1/ As a first step they 
were asked to assess their programs, identify their clients, and list 
potential indicators that would affect their clients’ views "of how 
well they were being served. As the activities are quite diverse, 
this produced a rather lengthy list of potential indicators which "would, 
however, be of value to program managers. From the viewpoint of the 
Office, it was desirable to develop a much shorter list which yet repre- 
sented the range of programs. So the next step was to develop the 
shorter list which is presented in this report. 


STATINTL 


I, CONSIDERATIONS 


The Office of Personnel shares with other elements of Agency 
management’ the responsibilities for staffing, career development, and 
supporting employee morale. In the personnel field the ultimate test 
of management is the effectiveness of Agency personnel in the execution 
of Agency programs. The Office of Personnel alone cannot assure this 
effectiveness but poor personnel programs could do much to preclude it. 
Given the inevitable pressure of resource constraints and the consequent 
desire of management to assign the higher priority to the operating 
programs, it is imperative that a staff fumction, such as personnel, 
evaluates the effectiveness of its programs so that the available resources 
are well-used and so that the resources required for essential personnel 
functions may be requested and defended. 
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With this need in mind, it was possible to specify some criteria 
for developing indicators: eh, stat 

a. On a selective basis, represent a broad range of Office 
activities. 


b. Represent a significant portion (30-60%) of Office resources. 


c. Indicate not how much but how well in aspects that pertain to 
objectives (where they exist) or client satisfaction. , 


d. Identify indicators that are based on client feedback or that 
themselves would influence feedback. 


e. Identify indicators that pertain to significant resources 
(don't bother with activities that require few resources). 


f. Differentiate between factors controllable by OP and those 
controllable outside of OP. , 


g. Provide D/Pers some indicators that pertain to the interaction 
between OP and other elements of management (system indicators). 


h. Reject indicators that involve great additional expense in 
assembling data unless management perceives they are worthy. 


i. subject the indicators to revision after use if required. 


These criteria have been applied, although not rigidly, in developing 
the indicators presented in this report. 


A complicating factor for the Office of Personnel is its unusually 
broad clientele -- employees, components, Directorates, and the Agency 
as a whole. Some personnel services -- such as position classification -- 
may produce conflicting perceptions of value among the employee concerned, * 
his Directorate, and the Agency as a whole. It is important, and not 
necessarily easy, to identify the relevant client. 


IJ. SYSTEM INDICATORS 


The personnel side of Agency management already has an unusual 
two part tool that provides significant indicators of Agency performance 
and effectiveness in such key areas as placement, separations, promotions, 
training, EEO, and executive development. The elements of this tool are — 
the Annual Personnel Plan and the Personnel Development Plan, The 
monitoring of performance against expectation provides significant manage- 
ment insights. The allocation of responsibilities for performance is 
usually apparent. 
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Additional system indicators might be applied. Given the high 

cost of recruiting and the decentralized nature of hiring, the Director 
of Personnel may wish to monitor the ratio between the number of © 
applicants in process (over the course of time) to the number being hired. - 
A growth in the ratio might indicate matters requiring management 
attention. A decline in the ratio would be highly desirable and the 
cause should be identified and encouraged. Along with this indicator, 
it would be useful to set a unit-cost type of indicator, such as the 
average cost per EOD (determined by dividing total Personnel Input" 
costs* by the number of EODs). Such an indicator could be expressed 
in constant dollars to permit comparison of changes over time so that 
trends of increasing or decreasing cost might be identified. Since costs 
of recruiting different types of personnel (professionals, clericals, 
blacks . . .) tend to vary significantly, some disaggregation of the cost 

‘ data might be desirable. 


Much of the cost of maintaining the personnel record system is 
influenced by data errors in originating offices. Hence, it is useful 
to monitor the number of errors found in personnel transactions as a 
ratio to the total number and to take corrective action with major offenders. 
Over time reduction of the proportionate number of errors would be a 
positive indicator of a successful policy of turnback and training.. 


III. EFFECTIVENESS INDICATORS FOR oP 


Much of the work of OP is susceptible to workload measurement and 
therefore to unit-cost measurement. Such measures are basically 
quantitative. Client perception of the effectiveness of OP is affected 
as much by the quality of service provided as the quantity, so measures 
of effectiveness must reflect qualitative considerations as well. A 
failure to do so would create incentive to increase quantity at the 
expense of quality. ; . 


In the following discussion indicators of effectiveness are identified 
for each of the program areas. A deliberate effort has been made to focus- 
on a small number of indicators for the Office overall. Our success in 


developing indicators has been uneven, as indicated by the appended 
comments. - 


Program Area I. Personnel Input 
Indicator A. Recruiting 
Semiannually, by general occupational title, calculate the ratio of 


applicants placed into process against manpower requirements,as stated in 
the Advance Staffing Plan. 


*The total costs attributed to the resource area for ‘Personnel Input." 
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Background: As recruiting against requirements is a major 
responsibility of the Office of Personnel, the provision of an adequate 
flow of qualified applicants is a key objective. The Advanced Staffing 
Plan has some inadequacies as an operational statement of recruiting 
needs, but application of this indicator would generate improvements in ; 
the statement of requirements. Past experience suggests that requirements ~ 
normally can be filled if the proper number of applicants are put in 
process. On an annual basis, the current norm for professional/technical 
openings is 2.8 applicants in process per opening and for clerical, two 
for one. If the statement of requirements is adequate, application of 
the above indicator would help to identify for management attention 
problem areas in the recruiting effort. 


Critique: The indicator may be affected by lags between the 
establishnent of a requirement and recruiting against it. Thus the 
indicator may show no applicants in process even though the appropriate 
recruiting efforts have been started and may stimulate overcompensation. 
This problem is the basis for using a semiannual indicator rather than 
quarterly reporting. 


. A second problem is that of defining the occupational groups, which 
should be relatively comprehensive. 


A third problem is that applicants may be listed under occupational 
titles that differ from those of the jobs in which they are actually 
placed. . 


A fourth problem is that a successful effort to reduce the ratio of 
applicants in process to EODs (a system effectiveness indicator) might 
cause this indicator to signal an inadequate number of applicants in 
process. 


Indicator B. EEQ-Related Recruiting . 
On an annual basis, measure the number of Black and Hispanic-American 


professional/technical applicants placed into process against manpower - 
requirements stated in the Annual Personnel Plan. ° 


Background: This indicator would signal success or lack of success 
in EEO-related recruiting. The current norm is 2.8 applicants in process ~ 
to 1 required. 


Critique: The indicator suffers from several of the weaknesses of 
the preceding indicator. 


Program Area If. _Personnel Operations 
Indicator A. Reassignment 


This is a compound indicator consisting of four annual measures: 
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1. The number of emphovecs who have contacted SPD. convenes 
reassignment. . 


Z. The number subsequently reassigned. Veen 


3. The number of terminated employees (voluntary or involuntary) 
whose Peas S tere cases have been closed by SPD within one year. 


. 4, Number of employees who have contacted SPD coneenne 
reassignment and whose cases are still active. 


Background: The objective is to develop an indicator of the 
effectiveness of the internal market. Since many of the cases that 
contact SPD may fall into the problem category it is to be expected that 
there will. be many failures in the form of employees who eventually 

‘terminate. Successful reassignments alone would not be a satisfactory 
indicator, for successful counseling may induce some employees to remain 
on the job. 


Critique: The indicator is not clean in the sense that OP has 
control. Obviously successful reassignment would involve factors 
external to OP. 


The choice of a one-year cutoff for previous contact with SPD con- 
cerning reassignment is arbitrary. Experience may indicate that many 
terminations occur between 12 and 24 months after contact. 


- As a compound indicator, it may be difficult to interpret for 
management purposes. 


Indicator B. Retirement Benefits Counseling 


Relate (1) the number of valid complaints concerning misinformation 
or delays in the provision of benefits information to (2) the number of 
retirees. 


some 


Background: The objective is to provide a measure of the effective- 
ness of retirement counseling. As valid complaints provide an indicator 
of reduced effectiveness, the objective is to reduce the number of 

. complaints. (As some complaints may be without basis, it is necessary to 
‘sort out those complaints that can be substantiated from those that can 
not. ) / 

Critique: The indicator will be very sensitive to the determination 
of whether a particular complaint is valid or not. 
It is implicit that there is a close relationship between the number 


of errors or delays and the number of complaints. One can imagine 
circumstances when complaints would be more likely to be registered. 


5 
Approved For Release 2001/05/23 : CIA-RDP83T00573R000600030027-7 


Adriinintentuen Pete ccef Taq Bely 


a, 


. Administvetiva - Ecol Use Unly 


Approved For Release 2001/05/23 : CIA-RDP83T00573R0006Q0030027-7 


As there is a close correlation between the number of retirees 
and the number receiving counseling related to retirement, it is valid 
to use the number of retirees in this indicator. 

Bk isos 


Indicator C. Problem Counseling -- Alcohol Problems 


On an annual basis, compare the number of employees receiving OP 
counseling related to problems of alcohol with the number terminated who 
have received such counseling. 


Background: The objective is to provide an indicator of the 
effectiveness of problem-oriented counseling. If an employee with a 
problem whc has received counseling is unable to hack it, it represents 
a failure in several respects but not exclusively the responsibility of 
the counseling program. 


Critique: There is a time lag problem so that it would be possible 
in a year in which problem counseling is very effective to have many 
terminations which really result from previous years in which counseling 
was not so effective. 


Successful outcome in problem counseling would relate to inany 
factors that are beyond OP control. The same is true of failures. 


Program Area III. Manpower Management 


Indicator A. Position Grade Control 


On an annual basis compare the’ Agency's actual change in average 
position grade with the average grade projection contained in the Agency's 
budget. 


Background: The objective is to control the Agency's position 
average grade So that it remains within the projection contained in the 
budget -- which presumably would reflect programmatic and substantive 

requirements. , = 


Critique: The actual change in average position grade would be . 
sesnined by many factors outside of OP control, yet if that is under- 
stood the measure has value. 


In the event that the realized change does not equate to the budget 
projection, it would be useful to disaggregate the data enough to identify 
the major causes and’ the components contributing. 


Indicator B. Review of Positions 
Fa 


On an annual basis, compare the number of positions reviewed for 
proper grade classification against the total number of Agency positions. 
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Background: A lengthening time between reviews of components and 
of thelr positions significantly reduces the validity of position 
descriptions and classifications. It is necessary to monitor progress 
in reclassification to preclude this. . 

ae tod 

Critique: This is a quantitative measure, not a qualitative one, 
so it Toute be possible to improve the performance in the indicator by 
degrading quality of position review. 


Program Area IV. Benefits and Services 
‘Indicator A. Employee. Perceptions 


Employee surveys should include sections to elicit feedback concerning 
levels of benefits and services provided, priority interests for new or 
expanded benefits and services and the adequacy of the administration of 

’ such programs. 


- 


Background: Management interests are served when employees are aware 
of benefits and services available to them, are assured that management 
will listen to their concerns for benefits and services commensurate with 
their contribution to the Agency, and are reasonably satisfied with the 
way in which programs are administered. 


Critique: The linkages between benefits and services and motivation 
are not clearly established once certain minimum requirements for economic 
security and the working environment have been met. Surveys should be 
careful not to establish an expectation that there is open season for new 
or expanded services and benefits. . 


The level of adequacy of the administration of benefits and services 
Will be a function of the resources provided, which will be relatively 
limited. Hence, some negative feedback is to be expected. The important 
aspect would be a trend toward a perception of worsening administration 
(longer times to process, more errors in processing) .which management 
might seek to avoid. Bs ee 


Indicator B. Level of Effort Indicators 


Strictly speaking, these are not effectiveness indicators but might - 
provide early warning of trends that would cause adverse employee 
“perceptions. Examples would be measures of the average time to process a 
claim, the number of errors reported, the number of employees briefed as to 
the availability of benefits and services, the number of publications 
distributed and the size of their distribution. 


Background: Such indicators already exist and provide management 
with some idea of the adequacy of administration of benefits and services 
given the available level of resources. The additional need is for an 
indicator that might alert management if its judgment of adequacy is at 
odds with employees' perceptions. 
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( Indicator C. ‘Market Penetration 


Where employee enrollment for benefits is voluntary, it is possible 
to measure the extent to which employees take advantage of the opportunity. 
Thus, one could measure the average amount of UBLIC insurance outstanding . 
per employee. 


Background: Such an indicator would measure progress in informing 
and educating employees in the benefits available. 


Critique: Unlike the salesman on commission, the objective is not 
to sell but to inform. The final decision for such voluntary benefits 
should always be the employee's. What would a high value of UBLIC 
insurance per employee really indicate? 


. Program Area V. Personnel Record Keeping 
Indicator A. Errors 


By month and by year, measure the ratio of the number of errors in 
personnel transactions received for processing to the total number of 
transactions received. ; 


Background: A high error rate on personnel transactions received 
from components introduces unnecessary costs in OP operations. Keeping 
statistics on the error rate makes it possible to concentrate training 
where it is most needed and most effective. 


Critique: Though some cost is: involved in keeping the necessary 
records, aie indicator already has proven of value. 


Indicator B. Processing Time of Transactions 


Record in and out dates of transactions received for processing and 
report on a quarterly basis the average number of days required. 
Background: Processing of certain personnel transactions cannot 
be completed until the appropriate personnel folder is available. 
Delays often result because folders are not in file or required update 
material is not in the folder. Such delays create distortions in those 
. personnel statistics that are based on processing date rather than 
effective date. The indicator provides management a tool to see if the 
problem is worsening or being kept under control. 


Critique: Control of this indicator lies outside OP with those 
offices that have personnel folders out on loan. Improvement of the 


indicator would rest upon developing procedures to speed the,circulation 
of such folders. 
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Indicator C. Timely Statistical Reports 


Report on a monthly basis the average time Kee a request for 
data from SRB and the providing of such data. 


Background: The effectiveness of a central personnel statistical 
service is related a its ability to provide timely information. Otherwise, 


customers may seek their own data banks or oe may be made with 
imaccurate or incomplete information. 


‘Critique: The delays in providing information may be driven by 
a eds T programming and batch-processing in OJCS. The indicator would 

y @ very sensitive to the type of request received and would not : 
be stabte from month to month as the mix changes. Even so, the indicator 


should be of management value, and its improvement would help to forestall 
‘decentralization of the personnel data bank. 


Program Area VI. Internal Management s° fae ter esting | 
The degree of fulfillment of personnel goals for OP within the APP i 
and PDP provides an effectiveness indicator for internal management. The 


Office may wish to establish supplementary goals for career management 
. and counseling. 


ea 
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